Creating a load balanced spatial partitioning of a structured, diffusing system of particles

ABSTRACT

Techniques are disclosed for creating a load balanced spatial partitioning of a structured, diffusing system of particles. An exemplary method includes steps of determining a subset of a set of nodes within a given portion of the coordinate system intersected by a surface defined by points having a given distance from the surface of the given node; and mirroring the determined subset to at least another portion of the coordinate system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.11/392,405 and U.S. patent application Ser. No. 11/113,939, thedisclosures of which are incorporated by reference herein. Theseapplications have the same inventive entity and assignee as the presentapplication.

FIELD OF THE INVENTION

This invention generally relates to methods and systems for solvingn-body problems, such as classical molecular dynamics. Morespecifically, the invention relates to methods and systems for creatinga load balanced spatial partitioning of a structured, diffusing systemof particles.

BACKGROUND OF THE INVENTION

Over the last few years, unprecedented computational resources have beendeveloped, such as the BLUE GENE™ family of computers from theInternational Business Machines Corporation. See N. R. Adiga et al., “AnOverview of the Blue Gene/L Supercomputer,” Proceedings of the ACM/IEEESupercomputing 2002 Conference (SC'02), November 2002, the disclosure ofwhich is incorporated by reference herein. These resources may be usedto attack, among other issues, grand challenge life sciences problemssuch as advancing the understanding of biologically important processes,in particular, the mechanisms behind protein folding.

In order to address this goal, attention has been directed to creating aclassical molecular dynamics software package for long-time andlarge-scale molecular simulations. See F. Allen et al., “Blue Gene: AVision for Protein Science Using a Petaflop Supercomputer,” IBM SystemsJournal, vol. 40, no. 2, pp. 310-327, 2001, the disclosure of which isincorporated by reference herein.

Classical molecular dynamics is predominantly an n-body problem. Ann-body problem may be defined as the problem of finding, given theinitial positions, masses, and velocities of n bodies, their subsequentmotions as determined by classical mechanics. An n-body problem, forexample molecular dynamics (MD), proceeds as a sequence of simulationtime steps. At each time step, forces on particles, in MD atoms, arecomputed; and then the equations of motion are integrated to update thevelocities and positions of the particles. In order to compute theforces on the particles, nominally the force between each particle andevery other particle is computed, a computational burden of O(n²).

Practically speaking, molecular dynamics programs reduce the O(n²) bycutting off pair interactions at some distance. However for manyscientifically relevant molecular systems, the computational burden dueto the particle pair interactions remains large. In order to reachscientifically relevant simulation times, parallel computers arerequired to compute particle pair interactions rapidly. See B. G. Fitchet al., “Blue Matter: Approaching the Limits of Concurrency forClassical Molecular Dynamics,” Proceedings of the ACM/IEEE SC 2006Conference, Volume 11, Issue 17, November 2006, and B. G. Fitch et al.,“Blue Matter, an Application Framework for Molecular Simulation on BlueGene,” Journal of Parallel and Distributed Computing, vol. 63, pp.759-773, July 2003, the disclosures of which are incorporated byreference herein.

SUMMARY OF THE INVENTION

An illustrative embodiment of the invention provides a method ofcreating a load balanced spatial partitioning of a structured, diffusingsystem of particles. An exemplary method includes steps of determining asubset of a set of nodes within a given portion of the coordinate systemintersected by a surface defined by points having a given distance fromthe surface of the given node; and mirroring the determined subset to atleast another portion of the coordinate system.

Another illustrative embodiment of the invention includes an apparatus,comprising a memory and a processor coupled thereto, wherein theprocessor is operative to perform the steps of the method describedabove. Another illustrative embodiment of the invention includes acomputer program product comprising a computer usable medium havingcomputer usable program code embodied therewith. The computer usableprogram code comprising computer usable program code configured toperform the steps of the method described above.

Illustrative embodiments of the invention advantageously solve n-bodyproblems such as classical molecular dynamics by geometrically mappingsimulation space to a torus/mesh parallel computer's node space, therebyenabling the geometric n-body problem to be efficiently mapped to amachine with a fundamentally geometric processor space.

Illustrative embodiments of the invention create a load balanced spatialpartitioning of a structured, diffusing system of particles withpairwise interactions that is scalable to a very large number of nodesand has favorable communications characteristics, including well definedbounds on the number of hops and the number of nodes to which aparticle's position must be sent.

These and other objects, features, and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a part of a torus computer system that may be used topractice this invention.

FIG. 2 is a block diagram depicting an exemplary processing system inwhich inventive techniques may be implemented.

FIG. 3 shows a coordinate system with which an exemplary load balancingtechnique may be used.

FIG. 4 is a flowchart showing an exemplary load balancing technique.

FIG. 5 shows broadcast zones for two nodes.

FIG. 6 shows an example of an algorithm that allows deterministic,averaged interaction assignment.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

A preferred embodiment of the invention, described below in detail,provides a method and system for creating a load balanced spatialpartitioning of a structured, diffusing system of particles withpairwise interactions that is scalable to a very large number of nodesand has favorable communications characteristics including well definedbounds on the number of hops and the number of nodes to which aparticle's position must be sent.

Simulation space is divided up into equivalent volume elements. Nodepair interactions, which are interactions between the particles in apair of nodes, are assigned to one of the set of nodes defined by theintersection surfaces of the node pair; it should be noted that this setof nodes may be quite large.

Because pair interactions are cut-off beyond a defined radius R_(c), an“interaction surface” can be described by a group of nodes containing aportion of the surface of a suitable specified convex shape thatcontains the sphere of radius R_(b)=R_(c)/2 centered about a simulationvolume element. Such a convex shape may be a surface similar in shape tothe volume element itself, with rounded corners. Since the interactionsurface has a radius of R_(b) around the volume element, it alsocontains the space having a radius of R_(b) around each particle withinthe volume element. Accordingly, interactions between nodes may bemanaged rather than interactions between particles.

Accordingly, illustrative embodiments in accordance with inventiveprinciples provide a technique for eliminating imbalances caused bylocal fluctuations in interaction workload, this allowing productive useof more nodes than there are particles in the system. Such a techniquemay allow for increased efficiency in solving n-body problems, such asthose frequently encountered in molecular dynamics. This increasedefficiency may be achieved by, for example, balancing the loadassociated with particle interaction computations across computationalelements of the computer used to perform the simulation.

Although described primarily in terms of solving n-body problems onmesh/torus machines, any suitable computer or computer system may beused to implement the invention. In a preferred embodiment, theabove-mentioned BLUE GENE™ family of computers may be used. The BLUEGENE™/L parallel computational platform does interprocessorcommunication primarily using a 3D torus network. Processor space isdefined by the location of processors on the 3D torus network andcommunication costs between processors increase with distance. Themethods described herein enable the geometric n-body problem to beefficiently mapped to a machine with a fundamentally geometric processorspace, however these methods may also be applied to other networktopologies.

In the discussion herein, the following definitions/conventions areused:

The term “node” refers to the computational element that forms nodes inthe graph defined by the interconnection network. Any “node” may includeone or more processors. Any suitable processing unit or units may beused in the node, and it is not necessary to describe the detail of theprocessing units.

The term “simulation space” refers to the domain of the n-bodysimulation.

The term “node space” refers to the geometry of the interconnectionsbetween nodes of the machine. Typically the discussion herein refers toa machine with a three-dimensional torus/mesh interconnect geometry inwhich each node is connected to its six nearest neighbors and isaddressed by a triple of integers that represent its coordinates withinthe network. In a torus, periodic boundary conditions are implementedphysically by connecting the nodes on opposite “faces” of the machine.

FIG. 1 illustrates a part of a computer system or structure that may beused in the implementation of the present invention. More specifically,FIG. 1 shows a part of a Massively Parallel Supercomputer architecturein the form of a three-dimensional torus designed to deliver processingpower on the order of teraOPS (trillion floating-point operations persecond) for a wide range of applications. The Massively Parallelsupercomputer architecture, in an exemplary embodiment, may comprise65,536 processing nodes organized as a 64×32×32 three-dimensional toruswith each processing node connected to six (6) neighboring nodes.

In particular, FIG. 1 shows a torus comprised of eight nodes 120connected together by links or connections 130. It is clear to see howthis interconnect scales by increasing the number of nodes 120 along allthree dimensions. With current technology, this architecture can beleveraged to hundreds of teraOPS. As will be understood by those ofordinary skill in the art, other computer structures may also be used inthe practice of this invention.

FIG. 2 is a block diagram depicting an exemplary processing system 200formed in accordance with an aspect of the invention. System 200 mayrepresent the entire processing facility for use by the invention or apart thereof. System 200 may include a processor 210, memory 220 coupledto the processor (e.g., via a bus 240 or alternative connection means),as well as input/output (I/O) circuitry 230 operative to interface withthe processor. The processor 210 may be configured to perform at least aportion of the methodologies of the present invention, illustrativeembodiments of which are shown in the figures and described herein.

It is to be appreciated that the term “processor” as used herein isintended to include any processing device, such as, for example, onethat includes a central processing unit (CPU) and/or other processingcircuitry (e.g., digital signal processor (DSP), microprocessor, etc.).Additionally, it is to be understood that the term “processor” may referto more than one processing device, and that various elements associatedwith a processing device may be shared by other processing devices. Theterm “memory” as used herein is intended to include memory and othercomputer-readable media associated with a processor or CPU, such as, forexample, random access memory (RAM), read only memory (ROM), fixedstorage media (e.g., a hard drive), removable storage media (e.g., adiskette), flash memory, etc.

Furthermore, the term “I/O circuitry” as used herein is intended toinclude, for example, one or more input devices (e.g., keyboard, mouse,etc.) for entering data to the processor, and/or one or more outputdevices (e.g., printer, monitor, etc.) for presenting the resultsassociated with the processor. I/O circuitry 630 may comprise, forexample, one or more ports operative to receive and/or transmit one ormore data packets and/or one or more control messages.

Accordingly, an application program, or software components thereof,including instructions or code for performing the methodologies of theinvention, as described herein, may be stored in one or more of theassociated storage media (e.g., ROM, fixed or removable storage) and,when ready to be utilized, loaded in whole or in part (e.g., into RAM)and executed by the processor 210. In any case, it is to be appreciatedthat at least a portion of the components shown in the above figures maybe implemented in various forms of hardware, software, or combinationsthereof, e.g., one or more DSPs with associated memory,application-specific integrated circuit(s), functional circuitry, one ormore operatively programmed general purpose digital computers withassociated memory, etc. Given the teachings of the invention providedherein, one of ordinary skill in the art will be able to contemplateother implementations of the components of the invention.

A preferred embodiment in accordance with illustrative principles of theinvention efficiently solves the n-body problem by geometrically mappingsimulation space to a torus/mesh parallel computer's node space via aspatial decomposition. In a preferred embodiment, the mechanism fordistributing the work is a broadcast of particle positions followedpossibly by a reduction of forces on each particle to the node thatbroadcasted that particle (the home node).

A basic requirement of this method is the ability to broadcast data fromone node to a subset of all the nodes in the system. The requiredcollective operations can be constructed using point-to-pointcommunications however; hardware may offer features to enhance theperformance of these primitives such as multi-cast.

In a preferred mapping of simulation space to node space, simulationspace is divided into the same number of sub-volumes as there are nodes,and further, the simulation space preferably has the same extents(measured in voxels) in each dimension as node space (measured innodes).

A sparse surface set is a set of nodes chosen from the set of nodesintersected by the “interaction surface” described above. Each node oneach time step will broadcast to its sparse surface set, compute forces,then sum-reduce the force values on each particle. More specifically,two reciprocal communications operations are of interest:

-   -   Sparse surface set broadcast: given a convex shape in simulation        space, broadcast data from the node containing the center of        such a shape to a set of nodes intersecting a portion of the        surface of that shape.    -   Sparse surface set reduce: given a convex shape in simulation        space, theta-reduce data from a set of nodes intersecting a        portion of the surface of that shape to the root node containing        the center of the shape.        N-Body Spatial Decomposition for Load Balance

N-body problems in a periodic simulation space nominally have acomputational burden defined by the order O(N²) pairwise particleinteractions possible. In many applications, this computational burdenis reduced to O(N) by cutting off interactions between particlesseparated by more than a specified distance in periodic simulationspace.

A natural way to map a spatial problem on a spatial connected machine iswith a direct mapping, where the simulation space is uniformly assignedto processor space. For n-body systems (assuming that all pairinteractions have the same computational cost) with a uniform density ofparticles or where no cut-offs are imposed, this simple mapping willresult in a balanced load across the machine since each node or regionof nodes will have roughly the same interaction computation workload.

However, a direct mapping may result in load imbalance. Load imbalancemay be caused by uneven distribution of particles or differences in thetypes of load, such as those based on particle type or distance betweeninteracting particles. There are several schemes for load balancingwhich may utilized in conjunction with embodiments of the presentinvention. Examples of such schemes may be found in, for example, U.S.patent application Ser. No. 11/392,405 and U.S. patent application Ser.No. 11/113,939, the disclosures of which are incorporated by referenceherein.

Preferred Load Balancing Algorithm

A preferred load balancing method suitable for use with illustrativeembodiments of the present invention is described herein with referenceto the coordinate system shown in FIG. 3 and the flowchart shown in FIG.4. The load balancing method described herein is suitable for use with,for example, n-body problems such as those associated with moleculardynamics.

FIG. 3 shows a portion of a three-dimensional relative coordinate systemdefined on a lattice of nodes. This coordinate system has its origin onReference Node 300, which has x-y-z coordinates <0, 0, 0>. Thiscoordinate system will be used to illustrate certain definitions andconventions to be used in describing the method.

The coordinate system includes a Positive Octant (PO), defined herein asthe set of nodes having non-negative displacements from the origin(e.g., Reference Node 300) in all three dimensions. The portion of thecoordinate system shown in FIG. 3 includes 49 nodes within the PO: nodes300-306, which have coordinates ranging from <0, 0, 0> to <0, 6, 0>;nodes 310-316, which have coordinates ranging from <1, 0, 0> to <1, 6,0>; nodes 320-326, which have coordinates ranging from <2, 0, 0> to <2,6, 0>; nodes 330-336, which have coordinates ranging from <3, 0, 0> to<3, 6, 0>; nodes 340-346, which have coordinates ranging from <4, 0, 0>to <4, 6, 0>; nodes 350-356, which have coordinates ranging from <5, 0,0> to <5, 6, 0>; and nodes 360-366, which have x-y-z coordinates rangingfrom <6, 0, 0> to <6, 6, 0>.

A given node having coordinates <x, y, z> may be considered one of a setof up to eight “mirror nodes,” which are defined as the nodes havingcoordinates <±x, ±y, ±z>. It should be noted that a set of mirror nodeswill typically include only one node within each octant. Moreover, thisset of mirror nodes will include a node within each octant, unless oneor more of the x-y-z coordinates is zero. For example, node 323, whichhas coordinates of <2, 3, 0>, has mirror nodes including node 392, whichhas coordinates of <2, −3, 0>; node 394, which has coordinates of <−2,−3, 0>; and node 398, which has coordinates of <−2, 3, 0>. Node 303,which has coordinates of <0, 3, 0>, has only one mirror node shown inFIG. 3; namely, node 393, which has coordinates of <0, −3, 0>. It shouldbe noted that finding the mirror within the Positive Octant (or POmirror) for a given node merely requires taking the absolute value ofeach coordinate. For example, the PO mirror of a node with coordinatesof <−13, 4, −2> is <13, 4, 2>.

It should noticed that although the illustrative embodiment describedherein is implemented using a three-dimensional coordinate system, othercoordinate systems may be used in conjunction with illustrativeembodiments of the invention. For example, an illustrative embodimentmay be implemented using a two-dimensional coordinate system, in whichthe positive octant may be replaced by the positive quadrant. Likewise,other portions of the coordinate systems may be used instead of thepositive octant and/or positive quadrant included, but not limited to,other octants and/or quadrants within the coordinate system.

The Effective Cutoff, or Rc, is a specified distance from a given nodebeyond which pair interactions are cut off.

A Node Interaction Set is defined as the set of nodes which map to (orcontain) any simulation space within Effective Cutoff of a given node,typically the Reference Node. This set may be thought of as comprisingthe nodes interacting with those of the Reference Node.

PO-nodes refer to those nodes within both the Node Interaction Set andthe Positive Octant. In the coordinate system shown in FIG. 3, sphere380 has a radius equal to the Effective Cutoff. The PO-nodes shown inFIG. 3 include nodes 300 to 306, 310 to 316, 320 to 326, 330 to 336, 340to 345, 350 to 354 and 360 to 363. At least a portion of each of thesenodes is contained within sphere 380. The Node Interaction Set shown inFIG. 3 includes the PO-nodes as well as the mirrors of the PO-nodes,such as nodes 391-398.

A Full Broadcast Skin includes all those nodes intersected by a surfacedefined by all points having a distance of one-half of the EffectiveCutoff from the surface of the Reference-Node. In other words, the FullBroadcast-Skin is the set of nodes containing some portion of the locusof points exactly one-half of the Effective Cutoff from the surface ofthe reference nodes. The PO Full Broadcast Skin are the nodes on theFull Broadcast Skin of the Reference Node in the Positive Octant.

In the coordinate system shown in FIG. 3, sphere 385 has a radius equalto one-half of the Effective Cutoff (i.e., one-half of the radius ofsphere 380). Thus, the Full Broadcast Skin includes those nodes throughwhich sphere 385 passes. The Full Broadcast Skin includes the PO FullBroadcast Skin, which includes nodes 303, 313, 323, 322, 332, 331 and330, as well as the mirrors thereof, such as nodes 391-398.

A Template Sparse Broadcast-Skin is defined as the set of nodes whichthe Reference-Node must send particle positions to such that if everyother node sent particles to an equivalent relative Sparse Skin Set,every pair of particles within cutoff of the Reference-Node would berealize on at least one Sparse Broadcast-Skin node. A PO SparseBroadcast-Skin comprises those nodes in the Template SparseBroadcast-Skin set and the Positive Octant. As will be discussedhereafter, the method will first develop a PO Sparse Broadcast-Skinwhich then, based on 3D symmetry, will be mirrored to create a TemplateSparse Broadcast-Skin.

Turning now to the flowchart shown in FIG. 4, the method begins in step410 with the assignment of volume elements to nodes such there is aone-to-one correspondence between nodes and volume elements in thesimulation elements. This preferably is a “natural” mapping in which thesimulation space is divided into equally sized volume elements that mapsimulation space onto node space in a way similar to that which would beobtained via the ORB technique if load were uniformly distributedthroughout the simulation volume.

In step 420, a PO option table for the Reference-Node is created with anentry for every PO node. Each entry within this PO option table is a POoption list is created by taking each node in the intersection of theReference-Node's Broadcast Skin and the PO node's Broadcast Skin,finding that node's mirror in the Positive Octant and making an entry ifthat mirror node has not already been added.

In step 430, the PO option table is sorted by PO option list size indescending order.

In step 440, the nodes on the Reference-Node's PO Broadcast Skin arecanonically numbered so that these numbers may be used to identify bitfields in a compressed representation of a PO Sparse Broadcast Skin asselected nodes on that PO Broadcast Skin. In the example shown in FIG.3, the Reference-Node's PO Broadcast Skin comprises nodes 303, 313, 323,322, 332, 331 and 330, which have canonically numbered as “0” through“6,” respectively. One having skill in the art will understand thatother techniques may be used to scan representations of sparse skins inaddition to or instead of this bit array technique.

In step 450, a PO option list is formed for each for each PO optiontable entry. This PO Option list is defined as a Binary PO Option Listwhich is a bit field with 0 for bits representing nodes not in the listand 1 for nodes that are on the list based on the canonical numbering ofstep 440, where the canonical cell number gives the bit position in thebit field. This is the Binary PO Option Table.

In the example shown in FIG. 3, this Binary PO Option List is 1010101.This bit field has a “1” in the 0th, 2nd, 4th and 6th positions, and a“0” in the 1st, 3rd, and 5th positions. This indicates that, of thenodes canonically numbered “0” to “6,” the nodes numbered as “0,” “2,”“4,” and “6” are in the list and the nodes numbered as “1,” “3,” and “5”are not in the list.

In step 460, a search is performed of the smallest set of reference nodePO Broadcast Skin nodes which have at least one node intersection inevery PO Option table entry. Because the lists in this table will besubsets of the full broadcast skins defined by the geometries of thesimulation and node spaces, the result of this process is referred to asthe PO sparse skin table.

This is a systematic search, and it may be parallelized by dividing thesearch space across parallel nodes. Starting with the number of nodesthat is the minimum estimated to satisfy the condition of having atleast one node that intersects every PO Option table entry, iteratethrough all combinations of PO Broadcast Nodes. For example, if thereare 32 nodes in the Reference-Node PO Broadcast Skin and five nodes areestimated to be the minimum, begin scanning 32-choose-5. Each bitpattern is a Candidate PO Sparse skin solution and is a scan value. Foreach scan value, loop over the Binary PO Option Table matching Binary POOption Lists.

Many variants of this scheme outlined are possible including theinclusion of a weighting field in the option table that could becomputed, for example, from the actual count of particle pairs withinrange for each pair of nodes/volume elements. Algorithms applicable tothe “bin-packing” problem could be adapted for use in thisload-balancing problem.

In an illustrative embodiment, the scan value fails if binary ANDing thescan value with any Binary PO Option list results in 0 (zero). A zerothe PO Broadcast Skin pattern does not satisfy all the interactionrequirements. Because we sorted the PO Option Table according tooccupancy of PO Option Lists, the first checks will be of the mostconstrained node pairs, thereby saving work. The scan succeeds if a scanvalue can be ANDed with each Binary PO Option List with a non-zeroresult—this is a candidate solution.

More than one candidate solution may occur when scanning for a certainnumber of nodes in the Candidate PO Sparse skin solution. One may selectthe first successful candidate or keep scanning for other candidatesthat may have better distributions to minimize communication costs. Apossible criterion for selecting among multiple solutions may be definedin terms of more detailed network routing costs given the specificdistribution of the solution.

In step 470, once a candidate is chosen, the candidate PO Sparse Skin isused to create a Template Sparse Broadcast Skin (the sparse skin for thereference node) by mirroring the candidate into each of the 7 otheroctants. As discussed above, the mirroring operation produces at most 7additional coordinates from the original coordinate <x, y, z> in the POby generating all permutations of <±x, ±y, ±z>. This is the group of allmirrors in three dimensions.

In step 480, the Template Sparse Broadcast Skin is used to generate thesparse skin for every node in the simulation by considering eachsimulation node as the Reference-Node and the solution set nodes asrelative displacements from the Reference-Node.

Molecular Dynamics Decompositions Based on Ellipsoidal CommunicationsPrimitives Optimal Interaction Assignment Using Dynamic InteractionBalancing

Molecular dynamics as applied to biomolecular simulation of fixed sizedproblems on very large, geometric node spaces requires a simulationspace to node space mapping that, preferably, maintains as much localityas possible while balancing workload. This is so communication costs andcomputational costs are minimized in balance.

To do this, the minimal set of communicating nodes for a particularparticle must be identified; for example, using the method describedabove with reference to FIGS. 3 and 4. The minimal set of communicatingnodes is defined as all those nodes which contain any of the surface ofa specified convex volume containing the spherical volume of space (withradius greater than half the cutoff radius) centered about the particle.This defines a minimal volume in node space that ensures that thereexists at least one node for every pair of particles within cutoff thatcan calculate the interaction between that pair.

During each simulation step, each node sends all its particle positionsto each of its communication partners and receives from each a set ofparticle positions. A node may be assigned to evaluate the interactionbetween any pair of particles in the set of positions it has received.

Although pairs of particle positions in the simulation space separatedby close to the cutoff distance might be received by only a single nodecreating a definitive assignment, generally the positions of any givenpair of particles will be received by several nodes as shown in FIG. 5,any one of which may be assigned that pair's interaction.

FIG. 5 shows the broadcast zones 610 and 620 for two nodes superimposedon the spatial decomposition of the domain onto all nodes(two-dimensional view for simplicity). The nodes that contain portionsof the surface in simulation space defined by the surface of the volumedefined by the set of points within radius R_(b) of any voxel assignedto Node A are shown in one type of hatching 630 with a differenthatching 640 where the nodes also contain portions of the surface insimulation space defined by the surface of the volume defined by the setof points within Rb of any voxel assigned to Node B (which are markedwith one type of hatching 650 except where overlap occurs). Thebroadcast radius R_(b)>R_(c)/2 where R_(c) is the cutoff radius. Theinteraction between a particle stored on Node A and a particle stored onNode B can be computed on any of the nodes with cross-hatching.

Interaction assignment may be accomplished by any method, which resultsin the interaction between a pair of particles being done once on onenode. An example of a suitable algorithm is to specify that theinteraction will be done on the node that has received both particlepositions and has the minimum node id number. Such algorithms howeverhave load balancing characteristics which reflect the spatial mappingtechnique used.

In order to reduce micro load imbalance, dynamic deterministicinteraction assignment can be employed. These techniques allow rapidoverlapping regional averaging. To enable these techniques, each nodemust measure its current computational load and add this value to eachmessage it sends to its communication partners. When a node receives itsset of [load, position set] messages from its communicating partners, itnow has enough information to deterministically assign itself a set ofinteractions. The set of interactions each node can contribute to loadaveraging among its communication partners.

An example of an algorithm, illustrated in FIG. 6, which allowsdeterministic, averaged interaction assignment is as follows:

At step 710, receive a complete set of [load, position set] messagesfrom communicating partners. At step 720, for each pair of receivedmessages, create a canonical interaction list including the effects ofrange cutoffs. At step 730, for each pair of messages received,determine the set of nodes that also received this pair of messages. Atstep 740, using a deterministic algorithm, assign each interaction(appearing on the canonical interaction list) that could be computed bythe above set of nodes to one of those nodes.

In an illustrative embodiment, assignment could be done on a node basisas a bin-packing optimization problem to achieve good load balance. Foreach pair of nodes, the interaction for the particles contained withinthat pair is assigned to a node in the intersection of the SparseBroadcast Skin sets.

Other examples of deterministic algorithms which may be used include:

-   -   Using the loads reported on each of those nodes, assign        interactions (which have been ordered in some canonical way) to        attempt to equalize the new loads on each of the nodes.    -   Randomizing the assignment of interactions to the node set using        a hashing algorithm.    -   Assigning each interaction to the node owning the voxel that        contains the mid-point of the line connecting the interacting        pair of particles.

At step 750 of FIG. 6, do assigned particle pair interactions. At step760, send each computed interaction force back to the pair of nodes thatown the interacting particles.

Importantly, this algorithm costs only interaction computation. The costof returning the force should be added to reflect lower communicationcost of computing an interaction on a node which owns one of theparticles.

For the purposes of increased scope for load balancing or in cases wherecommunication costs are high relative to computation costs, it may beadvantageous to increase the simulation space radius determining thecommunication partner set beyond R_(c)/2 to increase the number of nodesavailable for load balancing.

As heretofore noted, the present invention can be realized in hardware,software, or a combination of hardware and software. Any kind ofcomputer/server system(s)—or other apparatus adapted for carrying outthe methods described herein—is suited. A typical combination ofhardware and software could be a general purpose computer system with acomputer program that, when loaded and executed, carries out therespective methods described herein. Alternatively, a specific usecomputer, containing specialized hardware for carrying out one or moreof the functional tasks of the invention, could be utilized.

The present invention, or aspects thereof, can also be embedded in acomputer program product, which comprises all the respective featuresenabling the implementation of the methods described herein, andwhich—when loaded in a computer system—is able to carry out thesemethods. Computer program, software program, program, or software, inthe present context mean any expression, in any language, code ornotation, of a set of instructions intended to cause a system having aninformation processing capability to perform a particular functioneither directly or after either or both of the following: (a) conversionto another language, code or notation; and/or (b) reproduction in adifferent material form.

Although illustrative embodiments of the present invention have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may bemade by one skilled in the art without departing from the scope orspirit of the invention.

What is claimed is:
 1. A method, comprising the steps of: determining asubset of a set of nodes within a given portion of a coordinate systemintersected by a surface defined by points having a given distance fromthe surface of a given node, wherein each node comprises a plurality ofparticles and wherein the given node comprises an origin point of thecoordinate system; and mirroring the determined subset of nodes to atleast another portion of the coordinate system, wherein mirroringcomprises mirroring each node in the determined subset of nodes, whichhas a given coordinate in a given octant, to a corresponding coordinatein at least one other octant of the coordinate system, wherein thedetermining and mirroring steps are repeated for each of a plurality ofnodes within the coordinate system to create a load balanced spatialpartitioning of a structured, diffusing system of particles with nodepair interactions that comprise interactions between particles in a pairof nodes.
 2. The method of claim 1, wherein the given portion of thecoordinate system is a quadrant of a two-dimensional coordinate system.3. The method of claim 1, wherein the given portion of the coordinatesystem is an octant of a three-dimensional coordinate system.
 4. Themethod of claim 1, wherein the given distance is one half of aneffective cutoff distance.
 5. The method of claim 1, wherein the nodeswithin the coordinate system correspond to equally sized volumeelements.
 6. The method of claim 1, wherein the step of determining asubset of nodes further comprises the step of: creating an option tablecomprising a set of entries representing respective nodes within anintersection of the set of nodes and a set of nodes within the givenportion of the coordinate system intersected by a surface defined bypoints having the given distance from the surface of another node withinthe given portion of the coordinate system.
 7. The method of claim 6,wherein the subset of nodes comprises a minimal set of nodes within thesubset of nodes each having at least one node intersection in eachoption table entry.
 8. An apparatus, comprising: at least one memory;and at least one processor coupled to the memory, the processor beingoperative to perform the steps of: determining a subset of a set ofnodes within a given portion of a coordinate system intersected by asurface defined by points having a given distance from the surface of agiven node, wherein each node comprises a plurality of particles andwherein the given node comprises an origin point of the coordinatesystem; and mirroring the determined subset of nodes to at least anotherportion of the coordinate system, wherein mirroring comprises mirroringeach node in the determined subset of nodes, which has a givencoordinate in a given octant, to a corresponding coordinate in at leastone other octant of the coordinate system, wherein the determining andmirroring are repeated for each of a plurality of nodes within thecoordinate system to create a load balanced spatial partitioning of astructured, diffusing system of particles with node pair interactionsthat comprise interactions between particles in a pair of nodes.
 9. Theapparatus of claim 8, wherein the given distance is one half aneffective cutoff distance.
 10. The apparatus of claim 8, wherein thestep of determining a subset of nodes further comprises the step of:creating an option table comprising a set of entries representingrespective nodes within an intersection of the set of nodes and a set ofnodes within the given portion of the coordinate system intersected by asurface defined by points having the given distance from the surface ofanother node within the given portion of the coordinate system.
 11. Theapparatus of claim 10, wherein the subset of nodes comprises a minimalset of nodes within the subset of nodes each having at least one nodeintersection in each option table entry.
 12. A computer program productcomprising a non-transitory computer readable storage medium havingcomputer usable program code tangibly embodied therewith, wherein whenexecuted by a computer, the computer usable program code is configuredto perform the steps of: determining a subset of a set of nodes within agiven portion of a coordinate system intersected by a surface defined bypoints having a given distance from the surface of a given node, whereineach node comprises a plurality of particles and wherein the given nodecomprises an origin point of the coordinate system; and mirroring thedetermined subset of nodes to at least another portion of the coordinatesystem, wherein mirroring comprises mirroring each node in thedetermined subset of nodes, which has a given coordinate in a givenoctant, to a corresponding coordinate in at least one other octant ofthe coordinate system, wherein the determining and mirroring arerepeated for each of a plurality of nodes within the coordinate systemto create a load balanced spatial partitioning of a structured,diffusing system of particles with node pair interactions that compriseinteractions between particles in a pair of nodes.
 13. The computerprogram product of claim 12, wherein the given distance is one half ofan effective cutoff distance.
 14. The computer program product of claim12, wherein the step of determining a subset of nodes further comprisesthe step of: creating an option table comprising a set of entriesrepresenting respective nodes within an intersection of the set of nodesand a set of nodes within the given portion of the coordinate systemintersected by a surface defined by points having the given distancefrom the surface of another node within the given portion of thecoordinate system.
 15. The computer program product of claim 14, whereinthe subset of nodes comprises a minimal set of nodes within the subsetof nodes each having at least one node intersection in each option tableentry.